Repository navigation
Portable KV-backed Geo lookup - #454
ChristianPavilonis wants to merge 24 commits into
Conversation
e4ede10 to
c155e5d
Compare
aram356
left a comment
There was a problem hiding this comment.
PR Review
Summary
This PR adds a standalone KV Geo prototype: a CSV importer, the p1 binary index, digest-validated async lookup through KvHandle, the design spec and the results record. The lookup algorithm holds up under a randomized differential check: 872,448 lookups compared against naive range containment, covering both families, object budgets of 512–4096 bytes and cache budgets of 0, 2 KiB and 1 MiB, with zero mismatches. The blocking issues are about keeping the crate buildable and the evidence accurate. Nothing in CI builds it, its pins break on the next root bump, several tests can't fail, and the results doc's provenance doesn't match this head.
Findings
Blocking
- 🔧 No CI coverage, and exact pins coupled to the root: a simulated root
serde_jsonbump makes this crate unresolvable while CI stays green (examples/geo-kv-prototype/Cargo.toml:8) - 🔧 Object cache effectively untested: a cache that stores nothing passes all 10 tests (
examples/geo-kv-prototype/src/index.rs:635) - 🔧 Overlap-rejection test can't fail: no build succeeds at 140 bytes, and deleting the overlap check still passes (
examples/geo-kv-prototype/src/index.rs:763) - 🔧 Results doc provenance and coverage claims don't match this head (
docs/superpowers/plans/2026-11-05-kv-geo-prototype-results.md:9)
Non-blocking
- 🤔 Cache hit skips the family check, so warm and cold resolvers can disagree (
src/index.rs:531) - 🤔 IPv4-mapped input misses (
src/index.rs:501) - 🤔 Generation ID isn't derived from content (
src/index.rs:274) - ♻️ Validation runs after publication (
src/index.rs:291,src/main.rs:55) - ♻️ Second SHA-256 stack (
Cargo.toml:20) - 🤔
loadandArtifactStorehave no tests (src/import.rs:149) - ⛏ I/O errors lose the path (
src/import.rs:132) - ⛏
oracle_mismatchesis a constant (src/main.rs:128) - 🌱 An interrupted write blocks every rerun (
src/store.rs:98) - 🌱 Before p1 is frozen:
- The manifest serializes
u128bounds as JSON numbers and each digest as a 32-element array. IPv6 bounds above 2^53 lose precision in JavaScript andjq, and each digest takes about 115 characters as an array versus 64 as hex. Hex strings fix both. - Tests match
anyhowerrors by substring, which doesn't give the bounded failure categories spec §9 asks for. A typed error enum would. Cache::getcallsorder.retainon every hit, which costs O(entries). A production cache needs O(1) LRU bookkeeping.Resolver::lookup(&mut self)can't be shared across concurrent requests as written.
- The manifest serializes
📌 Out of Scope
- Dependabot's cargo entry covers only
/. Standalone lockfiles such as this crate's get updated only when someone notices a root bump broke them. A tracking issue across the standalone crates would cover this.
CI Status
- PR checks: 18/18 pass at
c52bdaf3 - Local workspace gates (fmt, clippy, test, feature check, spin wasm32): PASS
- Local prototype checks: fmt PASS, tests PASS (10/10), clippy
-D warningsPASS, release build PASS,cargo checkforwasm32-unknown-unknownandwasm32-wasip1PASS - None of the workspace gates or PR checks compile this crate.
How the findings were checked
- Mutation testing:
cargo mutantsonindex.rs,import.rsandstore.rsproduced 243 mutants: 146 caught, 71 missed, 20 unviable, 6 timeouts. - Probe tests in a scratch copy covered the cache/family divergence, IPv4-mapped input, the minimum budget and the randomized differential check.
- CLI run: end-to-end
build/verify/lookupon synthetic Country-shaped CSV.
dhruv8sh
left a comment
There was a problem hiding this comment.
PR Review
Summary
Standalone, opt-in geo-KV prototype (own [workspace], no production impact) plus design/results docs. Lookup algorithm looks correct and is well validated; only non-blocking notes.
😃 Praise
- Lookups are verified against a plain containment search over the original, unmerged source networks (
import.rs:224-236).XKis kept with acore_country_supportedflag rather than widening or dropping it, and the lookup path uses onlyKvHandlewithblock_ontests.
Findings
Non-blocking
- 🤔 Generation check after write (
src/main.rs:55) — inline - 🤔 O(n) cache hit bookkeeping (
src/index.rs:427) — inline - ⛏ Silent edition fallback (
src/import.rs:150) — inline - ⛏ Manifest size unbounded on read (
src/index.rs:466) — inline - 🌱 Prototype not exercised by CI: its own
[workspace]means root CI never builds or tests it, so it can rot asedgezero-corechanges. Consider acargo test --locked --manifest-path examples/geo-kv-prototype/Cargo.tomlstep or a follow-up issue. - 📝 Results doc predates the toolchain bump: it records Rust 1.95.0 and base
72ceed7e; the repo is now on 1.98.1 and the measurements weren't rerun. Keep that caveat visible.
CI Status
- fmt: PASS
- clippy (prototype,
-D warnings): PASS - tests (prototype): PASS (10 passed, 0 failed)
Fastly reads geo_lookup for the client address and Cloudflare reads the request's cf metadata. Both are always selected and pass their outcome through dispatch_admitted_with_geo, so public extensions still cannot replace it. Platform values that fail validation, such as Cloudflare's XX and T1 placeholders, report missing metadata rather than an invalid assertion. GeoSource and GeoTrustBasis gain platform variants and become non_exhaustive; fastly and cloudflare are reserved provider IDs.
[adapters.axum.geo] carries provider, alb_peer_cidrs and origin_credential_header_name defaults, each overridden by the matching EDGEZERO__ADAPTER__GEO__* setting. Precedence is with_geo, then settings, then the manifest, then off, and misconfiguration fails startup instead of falling back. The dev run_app reads settings only. geo_capability() stays explicit-only and geo_capability_for::<A>() reports the runner's view.
#517 (10ed380) tops ChristianPavilonis's line on #439: - #510, the KvStore record contract with DynamoDB and celld providers on Axum (#472, #473); - #454, portable KV-backed geo; - #514, country and region through ctx.geo() on Cloudflare and Spin (#480, #486); - #517, the portable ClientInfo contract with its Fastly, Axum, Cloudflare and Spin mappings (#475, #90, #91, #92, #485). #517 also merged #469 and #502 itself, so its tree contains everything this branch had. The only conflicts were JA4 lines that #517 refactors on top of #469; this merge takes #517's version, and its tree is identical to #517's.
What this PR delivers
This replaces the prototype-only serving story with opt-in, managed Axum KV Geo. The portable reader takes a checked IP and uses the existing
KvHandleinterface and a pinned immutable generation. It has no CSV, MMDB, native SDK, or backend-specific lookup dependency. Off remains the default; protected CloudFront and supplied-provider behavior are preserved.The standalone operator tool builds GeoLite City or Country CSV artifacts, publishes with read-back and manifest-last ordering, and verifies the full tree through the serving binding against an independent CSV oracle. It supports ordinary stock celld and DynamoDB KV. The serving identity is read-only; the publisher can retire prior generations by inventory. No new service API, storage provisioning, or EC binding is required.
docs/superpowers/specs/2026-11-05-kv-geo-design.mddocs/guide/native-geo.mdexamples/geo-kv-prototype/README.mdReview and landing order
This branch still targets #510 (
feat/kv-record-contract). It includes the #439 provider foundation and a local reconciliation of #502. Please confirm that #439 lands first, that #502 and #510 are reconciled into the target before #454 lands, and that #454 need not carry their changes as its own feature diff. The PR base has not been changed.Tracks #438. Maintainer decision requested before closure: accept a full-tree verification through the read-only serving binding before rollout, plus pinned manifest and nonempty root checks before Axum readiness, in place of the MMDB-specific malformed-file and mount checks. A required descendant missing or corrupt at request time is a provider failure, not a geographic miss or fallback.
Evidence
The local workspace run passed 3,051 tests with 9 existing ignores, alongside strict Clippy, feature checks, and the Spin WASM build. The earlier CLI test stall reproduces on both base and candidate only with terminal stdin. The workspace run used noninteractive stdin without adding any skips. The final standalone default and all-feature suites passed after the object-size decision. The Axum native feature lint matrix passed locally after the compiled-out fixture fixes.
The combined backend proof passed on stock celld and DynamoDB Local, including full-tree readback, managed Axum requests, and 4,096 IPv4/IPv6 oracle comparisons per backend and object size. The same 16 KiB artifacts avoided repeat KV reads in the sampled workload on both backends, so 16 KiB is now the default. 64 KiB remains an explicit option for fewer objects and fewer cold path reads. The report records which binaries were measured. Hosted IAM, production performance and durability are not claimed. No licensed source or derived data is checked in.
Trusted Server integration, deployment, and tags are out of scope for this PR.